The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations

Background: Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM.Methods: a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9.Results: The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a smaller SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2.Conclusions: An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use

Using Differential Item Functioning to evaluate potential bias in a high stakes postgraduate knowledge based assessment

BACKGROUND: Fairness is a critical component of defensible assessment. Candidates should perform according to ability without influence from background characteristics such as ethnicity or sex. However, performance differs by candidate background in many assessment environments. Many potential causes of such differences exist, and examinations must be routinely analysed to ensure they do not present inappropriate progression barriers for any candidate group. By analysing the individual questions of an examination through techniques such as Differential Item Functioning (DIF), we can test whether a subset of unfair questions explains group-level differences. Such items can then be revised or removed. METHODS: We used DIF to investigate fairness for 13,694 candidates sitting a major international summative postgraduate examination in internal medicine. We compared (a) ethnically white UK graduates against ethnically non-white UK graduates and (b) male UK graduates against female UK graduates. DIF was used to test 2773 questions across 14 sittings. RESULTS: Across 2773 questions eight (0.29%) showed notable DIF after correcting for multiple comparisons: seven medium effects and one large effect. Blinded analysis of these questions by a panel of clinician assessors identified no plausible explanations for the differences. These questions were removed from the question bank and we present them here to share knowledge of questions with DIF. These questions did not significantly impact the overall performance of the cohort. Group-level differences in performance between the groups we studied in this examination cannot be explained by a subset of unfair questions. CONCLUSIONS: DIF helps explore fairness in assessment at the question level. This is especially important in high-stakes assessment where a small number of unfair questions may adversely impact the passing rates of some groups. However, very few questions exhibited notable DIF so differences in passing rates for the groups we studied cannot be explained by unfairness at the question level

Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations

Background: The UK General Medical Council has emphasized the lack of evidence on whether graduates from different UK medical schools perform differently in their clinical careers. Here we assess the performance of UK graduates who have taken MRCP( UK) Part 1 and Part 2, which are multiple-choice assessments, and PACES, an assessment using real and simulated patients of clinical examination skills and communication skills, and we explore the reasons for the differences between medical schools. Method: We perform a retrospective analysis of the performance of 5827 doctors graduating in UK medical schools taking the Part 1, Part 2 or PACES for the first time between 2003/2 and 2005/3, and 22453 candidates taking Part 1 from 1989/1 to 2005/3. Results: Graduates of UK medical schools performed differently in the MRCP( UK) examination between 2003/2 and 2005/3. Part 1 and 2 performance of Oxford, Cambridge and Newcastle-upon-Tyne graduates was significantly better than average, and the performance of Liverpool, Dundee, Belfast and Aberdeen graduates was significantly worse than average. In the PACES ( clinical) examination, Oxford graduates performed significantly above average, and Dundee, Liverpool and London graduates significantly below average. About 60% of medical school variance was explained by differences in pre-admission qualifications, although the remaining variance was still significant, with graduates from Leicester, Oxford, Birmingham, Newcastle-upon-Tyne and London overperforming at Part 1, and graduates from Southampton, Dundee, Aberdeen, Liverpool and Belfast underperforming relative to pre-admission qualifications. The ranking of schools at Part 1 in 2003/2 to 2005/3 correlated 0.723, 0.654, 0.618 and 0.493 with performance in 1999-2001, 1996-1998, 1993-1995 and 1989-1992, respectively. Conclusion: Candidates from different UK medical schools perform differently in all three parts of the MRCP( UK) examination, with the ordering consistent across the parts of the exam and with the differences in Part 1 performance being consistent from 1989 to 2005. Although pre-admission qualifications explained some of the medical school variance, the remaining differences do not seem to result from career preference or other selection biases, and are presumed to result from unmeasured differences in ability at entry to the medical school or to differences between medical schools in teaching focus, content and approaches. Exploration of causal mechanisms would be enhanced by results from a national medical qualifying examination

"New Regionalism" and Development of Transborder Cooperation in European Union (A prospective analyses)

2005/2006This paper intends to be a prospective analysis of a phenomenon that is not new in Europe, but it is not even old: the cooperation and development of European regions. It reports on the results of research to explore a range of attempts to develop new regional forms in European Union, and considers the degree to which they accord to conceptualisations of the “new regionalism” and accounts of the changing territorial structure of the state. It highlights the array of new regional configurations which now extends across the territory of the European Union, discussing the influence exerted by the growth of interest in European spatial planning over the course of the 1990s and considering the degree to which readings of new regionalist rhetoric have informed both the creation and substance of a number of recently conceived regional entities. The structure of this thesis is formed by six chapters that develop one central idea of the situation of regions in European Union with particular interest on the “new regionalism” and development of trans-border cooperation in this part of the word. Therefore, we start the research paper by presenting in the first chapter - the Introduction - the main objectives of this study and the main questions that this thesis tries to answer. The second chapter - Theories of European Regional Policy – presents some of the most important theories about the concept of Regional Policy in European Union, that is also the most important mechanism that affects and tries to develop the cooperation between European regions, and the approach of this concept by different scholars all over the EU and not only. This chapter is structured in three parts: the Classical Theories of European Regional Policy in the context of European Union Policies, the New Theories and Proposals Regarding the European Regional Policy and the Cross-border Cooperation and European Regional Policy. These three ways of looking at the European Regional Policy are connected and try to approach this complex argument from more perspectives, as it can be observed in the pages of this chapter. In the chapter number 3 - The European Regional Policy and its Instruments- we are going forward into the analysis of the regional policy in European Union and try to find more connections between the regions from this perspective. In this sense, this chapter has three parts: The European Regional Policy - from the beginning to our days-, The Instruments of the European Regional Policy: A. Structural Funds and The Instruments of the European Regional Policy: B. INTERREG Programmes. We noticed, following this research, that not only the European regional policy is present in the daily life of European citizens, but also it is very active, and from its beginnings to our-days had more and more influence in the development of European regions. This was and is possible with the help of its instruments, the most important of these being presented in this chapter. Following the same framework, the next chapter, number 4 - The Theories of Regionalism in the European Union – wants to approach the issue of regional cooperation also from a more formal point of view. In this sense, the chapter has three parts: The Regionalism and the “New Regionalism” in the European Union, The Analytical Framework and the Trans-border Regions and the “New Regionalism”. This structure helps understanding the concept of regionalism and, more than that, seeing the connection between European regionalism, the European regional policy and the main actors in this field, the European regions. At this point of the study, it is necessary to give some examples of how it is working and how could work these theoretical aspects in practice. The chapter number 5 - The Analysis of Two Case-study of Regionalism in European Union – brings to our attention two examples of European regionalism, one from the beginning of the application of this concepts: A Case-study of “Old Regionalism” in European Union - Cross-border Cooperation: Italy and Austria -, and on the other side, an example of a more recent type of this phenomenon in the European Union, that is: A Case-study of “New Regionalism” in the European Union – Trans-national Cooperation North West Europe (NWE). The chapter concludes, in the third part: The Results of the Analysis between these Two Types of Regionalism in the European Union, by presenting some similarities of these two types of regionalism and the advantages and disadvantages that result from the application of this kind of regional cooperation. The final chapter, number 6 – Conclusions and Next Steps, presents the conclusions and some possible ways of action in the future, in the field of cooperation between the European regions, so the chapter is structured in two parts: Conclusions and Next Steps. The thesis concludes with a list of references that we used in the research. This part is the Bibliography. It is divided in the following parts: General Papers, Special Papers, Studies-Articles-Publications, European Union Documents and Electronic sources. For a better understanding of the arguments in discussion, we considered necessary to annex three documents: Cross-border Cooperation (ERDF) Maps in European Union, Trans-national Cooperation Areas Maps in the European Union and Nomenclature of Territorial Units for Statistics (NUTS). This represents the situation of the regions in the European Union. Within this structure, the thesis wants to put in light the importance and the role that the regions play in the European Union. The analysis of this kind of aspects could bring a better understanding of where and how the future will be for the regions in the European Union.Questa tesi vuole essere un’analisi prospettica di un fenomeno che non è nuovo in Europa, ma nemmeno conosciuto da tanto tempo: la cooperazione e lo sviluppo delle regione europee. Questo lavoro riporta i risultati della ricerca che ha tentato di analizzare in modi diversi le nuove forme regionali nell’Unione Europea, considerando in quale grado gli esperti sono d’accordo nel dare una definizione al “nuovo regionalismo” e nel rilevare le modifiche strutturali a livello territoriale dello stato. Nello stesso tempo, si sottolinea la moltitudine di nuove configurazioni regionali che adesso si possono individuare sul territorio dell’Unione Europea, esaminando l’influenza esercitata da un maggior interesse per la pianificazione spaziale - territoriale a partire dagli anni ’90, anche in considerazione del grado in cui le nuove teorie retoriche regionaliste hanno influito sulla creazione e sulla sostanza di nuove forme regionali che sono state create di recente. La struttura di questa tesi si basa su sei capitoli, ognuno dei quali prova a sviluppare da un punto di vista diverso, un’idea centrale della situazione delle regioni nell’Unione Europea, con un accento particolare sul fenomeno del “nuovo regionalismo” e lo sviluppo della cooperazione transfrontaliera in questa parte del mondo. In questo senso si è dato inizio a questo lavoro di ricerca, presentando nel primo capitolo, l’Introduzione, i principali obiettivi di questa tesi e le questioni più importanti per le quali si vogliono trovare delle risposte. Il secondo capitolo – Le teorie della politica regionale europea – presenta alcune delle più importanti teorie che riguardano il concetto della Politica Regionale nell’Unione Europea, che è nello stesso tempo il meccanismo principale che influisce e che prova a sviluppare la cooperazione tra le regione europee. In esso si mettono anche in evidenza le teorie di diversi studiosi europei e non solo, che hanno analizzato questo argomento. Il capitolo è strutturato in tre parti: Le teorie classiche della politica regionale europea nel contesto delle politiche europee; Le nuove teorie e le proposte che riguardano la politica regionale europea; La cooperazione transfrontaliera e la politica regionale europea. Questi tre modi di avvicinarsi al tema della politica regionale europea sono collegati tra di loro e provano a studiare questo fenomeno complesso da svariate prospettive, come si può desumere dalle pagine di questo capitolo. Nel terzo capitolo – La politica regionale europea e i suoi strumenti – si è cercato di procedere nell’analisi della politica regionale nell’Unione Europea e di trovare più connessioni tra le regioni da questo punto di vista. In tal senso, questo capitolo si sviluppa in tre parti: La politica regionale europea – dalle origini fino ad oggi; Gli strumenti della politica regionale europea: A) i fondi strutturali; Gli strumenti della politica regionale europea: B) i programmi INTERREG. Nel corso di questa ricerca abbiamo notato che non solo la politica regionale europea è presente nella vita quotidiana dei cittadini europei, ma che, da quando ha avuto inizio questa sua presenza e fino ai nostri giorni, è stata ed è molto attiva, avendo un’influenza crescente nello sviluppo delle regioni europee. Ciò è stato ed è tuttora possibile grazie ai suoi strumenti, i più importanti dei quali sono stati presentati e studiati in questo capitolo. Seguendo la stessa costruzione, il successivo capitolo quattro – Le teorie del regionalismo nell’Unione Europea – vuole approfondire l’argomento della cooperazione regionale anche da un punto di vista più formale. In tal senso, il capitolo è suddiviso in tre parti: Il regionalismo e il “nuovo regionalismo” nell’Unione Europea; La struttura analitica; Le regioni transfrontaliere e il “nuovo regionalismo”. Questa struttura aiuta a capire il concetto di “regionalismo” e, soprattutto, prova a dimostrare le connessioni esistenti tra il regionalismo europeo, la politica regionale europea e i principali attori in questo campo, le regioni europee. A questo punto della ricerca, si è ritenuto necessario dare qualche esempio di come funzionano e come potrebbero funzionare in pratica questi concenti teorici. Il capitolo quinto – L’analisi di due casi di studio del regionalismo nell’Unione Europea – sottopone alla nostra attenzione due esempi di regionalismo europeo, di cui uno può rappresentare le prime fasi nell’applicazione di questi concetti teorici: Un caso di studio del “vecchio regionalismo” nell’Unione Europea – Cooperazione transfrontaliera: Italia e Austria; mentre l’altro caso fornisce un esempio di un tipo di regionalismo manifestatosi più di recente nell’Unione Europea: Un caso di studio del “nuovo regionalismo” nell’Unione Europea – Cooperazione transnazionale: Nord-Ovest Europa (NWE). Il capitolo si conclude con una terza parte: I risultati dell’analisi tra questi due tipi di regionalismo nell’Unione Europea. In questa parte si è cercato di rappresentare alcune delle similitudini risultanti dall’analisi, nonché i vantaggi e gli svantaggi derivanti dall’applicazione pratica di questo tipo di cooperazione regionale. Nell’ultimo capitolo, numero sei, – Conclusioni e Azioni Future, sono esposte le conclusioni e le possibili direzioni future d’azione nel campo della cooperazione tra le regioni europee. Questo capitolo è strutturato in due parti: Conclusioni; Azioni Future. La tesi si conclude con un elenco delle risorse bibliografiche utilizzate per questa ricerca. Questa parte, denominata Bibliografia, è strutturata nelle parti seguenti: Libri Generali; Libri di Specialità; Studi, Articoli, Pubblicazioni; Documenti dell’Unione Europea; Fonti Elettroniche. Per una migliore comprensione degli argomenti affrontati in questa tesi, si è ritenuto necessario di allegare tre documenti: Le cartine della Cooperazione Transfrontaliera nell’Unione Europea; Le cartine della Cooperazione Transnazionale nell’Unione Europea; La Nomenclatura delle Unità Territoriali Statistiche (NUTS). Questi documenti rappresentano l’attuale situazione delle regioni nell’Unione Europea. Con questa impostazione, la tesi vuole mettere in evidenza l’importanza e il ruolo che le regioni giocano nell’Unione Europea. L’analisi di questo tipo di argomenti può portare a una migliore comprensione circa la configurazione del futuro per le regioni nell’Unione Europea.XIX Cicl

Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations

Abstract Background The UK General Medical Council has emphasized the lack of evidence on whether graduates from different UK medical schools perform differently in their clinical careers. Here we assess the performance of UK graduates who have taken MRCP(UK) Part 1 and Part 2, which are multiple-choice assessments, and PACES, an assessment using real and simulated patients of clinical examination skills and communication skills, and we explore the reasons for the differences between medical schools. Method We perform a retrospective analysis of the performance of 5827 doctors graduating in UK medical schools taking the Part 1, Part 2 or PACES for the first time between 2003/2 and 2005/3, and 22453 candidates taking Part 1 from 1989/1 to 2005/3. Results Graduates of UK medical schools performed differently in the MRCP(UK) examination between 2003/2 and 2005/3. Part 1 and 2 performance of Oxford, Cambridge and Newcastle-upon-Tyne graduates was significantly better than average, and the performance of Liverpool, Dundee, Belfast and Aberdeen graduates was significantly worse than average. In the PACES (clinical) examination, Oxford graduates performed significantly above average, and Dundee, Liverpool and London graduates significantly below average. About 60% of medical school variance was explained by differences in pre-admission qualifications, although the remaining variance was still significant, with graduates from Leicester, Oxford, Birmingham, Newcastle-upon-Tyne and London overperforming at Part 1, and graduates from Southampton, Dundee, Aberdeen, Liverpool and Belfast underperforming relative to pre-admission qualifications. The ranking of schools at Part 1 in 2003/2 to 2005/3 correlated 0.723, 0.654, 0.618 and 0.493 with performance in 1999–2001, 1996–1998, 1993–1995 and 1989–1992, respectively. Conclusion Candidates from different UK medical schools perform differently in all three parts of the MRCP(UK) examination, with the ordering consistent across the parts of the exam and with the differences in Part 1 performance being consistent from 1989 to 2005. Although pre-admission qualifications explained some of the medical school variance, the remaining differences do not seem to result from career preference or other selection biases, and are presumed to result from unmeasured differences in ability at entry to the medical school or to differences between medical schools in teaching focus, content and approaches. Exploration of causal mechanisms would be enhanced by results from a national medical qualifying examination.</p

Additional file 1: of Using differential item functioning to evaluate potential bias in a high stakes postgraduate knowledge based assessment

High resolution images of the plots contained in the paper. (ZIP 2058 kb

Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations-2

<p><b>Copyright information:</b></p><p>Taken from "Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations"</p><p>http://www.biomedcentral.com/1741-7015/6/5</p><p>BMC Medicine 2008;6():5-5.</p><p>Published online 14 Feb 2008</p><p>PMCID:PMC2265293.</p><p></p

Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations-0

Ve average values). Error bars indicate ± 1 SE and because sample sizes are large (typically of the order of 500 and over 5000 in the case of London), error terms are small (see the text).<p><b>Copyright information:</b></p><p>Taken from "Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations"</p><p>http://www.biomedcentral.com/1741-7015/6/5</p><p>BMC Medicine 2008;6():5-5.</p><p>Published online 14 Feb 2008</p><p>PMCID:PMC2265293.</p><p></p

Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations-1

M. The width of paths is proportional to the path coefficient. The saturated model allowed all variables to the left of a variable to have a causal influence on that variable and non-significant paths were removed until paths remaining were significant with < 0.05. Paths not shown as causal arrows did not reach significance with < 0.05.<p><b>Copyright information:</b></p><p>Taken from "Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations"</p><p>http://www.biomedcentral.com/1741-7015/6/5</p><p>BMC Medicine 2008;6():5-5.</p><p>Published online 14 Feb 2008</p><p>PMCID:PMC2265293.</p><p></p